Skip to content

Simplify AI Gateway skills and align portable plugin workflows - #38

Closed
Paul Yuknewicz (paulyuk) wants to merge 20 commits into
feature/apim-migration-native-sync-20261105from
paulyuk-ai-gateway-skill-improvements
Closed

Paul Yuknewicz (paulyuk) wants to merge 20 commits into
feature/apim-migration-native-sync-20261105from
paulyuk-ai-gateway-skill-improvements

Conversation

@paulyuk

@paulyuk Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Azure AI Gateway's plugin should complete the documented CLI and data-plane scenarios reliably, without unnecessary discovery or large instruction loads. This PR removes known obstacles to that goal and prepares repeatable evaluation. It is an A-targeting candidate, not an A-rated result. Local checks establish packaging and deterministic behavior, not reliable live gateway behavior or performance close to the CLI.

The corrected candidate passed 21/21 in one fresh frozen trial, then 16/18 originals and 3/3 development variants, or 19/21, in the next unchanged trial. Bootstrap passed both. Both trials reached inference throughout, passed native syntax 21/21 and retained complete deny-all evidence with zero tool calls/events/denials. There was no identified setup failure. The latest eval rejects correctly scoped alias lists and an equivalent composite-counter label. Separately, the alias plan omits the management skill's same-name gateway-family precaution; the current grader permits that read but does not independently require it. A comparison fix alone must not certify the missing precaution.

The bootstrap input explicitly limits optional import to one selected existing deployment; separately approved whole-account onboarding remains distinct. Case-only identity normalization retains exact operation, role, principal and scope checks. The earlier ambiguous-request output and offline replay remain historical, not fresh corrected-input evidence. The three historical holdout tasks are repeatedly inspected development variants, not genuine held-out validation.

Paid execution is stopped. Two of the three latest approved frozen invocations were executed; one remains unused and cannot establish three consecutive green trials after these findings. Next, correct the scoped-read/composite-partition comparisons offline, preserve their negative safety tests and resolve the alias family-preflight omission before restarting consistency evidence. No post-run rubric or input change relabels the fresh 19/21 score. The detailed historical journals now have human TL;DR openings, edited in place without deleting evidence.

Follow-up to #32, targeting feature/apim-migration-native-sync-20261105 so these improvements can be reviewed and merged into the original PR first. The original branch and native-stack metadata are unchanged.

What changes

  • Make the existing management and consumer skills task-scoped and CLI-first. Move detailed consumer instructions into a progressive reference; keep the runtime-only path free of ARM, login, release checks, and unrelated inventories. Explanations with sufficient supplied evidence skip management setup/discovery. When an optional gateway-root read is relevant, prefer the fully typed native show --ids; root inspection is not mandatory.
  • Support explicitly requested resource-group bootstrap and selected Foundry onboarding, with separate approval for system identity and account-scoped Foundry User effects. Preserve migration safeguards, destructive/access/secret approvals, sibling preservation, and partial-failure reconciliation.
  • Resolve hosted-doc/CLI differences through the selected gateway's verified alias, endpoint, and authentication contract, rather than automatic header or identity fallback. Include bounded data-plane response checks and native b22 syntax/applicability guidance.
  • Add the portable Agent Plugins root manifest and thin Copilot adapters; synchronize existing host manifests and marketplace versions. Document supported formats without claiming awesome-copilot marketplace publication or untested host interoperability. Keep the four public skill names and boundaries, with migration bundled and feedback presented as a support utility. All six compatibility commands route to one authoritative skill recipe rather than repeating workflows across Markdown and Gemini adapters. Move development regressions outside runtime skill directories without changing executable helpers, catalogs or schemas. The portable manifest has no standard exclusion field, so repository downloads may still contain development files.
  • Add 18 Waza regression fixtures and three separately selected development variants, pinned schemas, typed output-contract shapes, native-help provenance, and deterministic positive/negative checks. Correct reviewed URL, identity, scoped-ID, assessment-phase and bounded-readiness rubric defects. The active quality contract checks behavior-relevant facts and approved versus approval-pending command plans without arbitrary decision/gate labels. It accepts lossless structured identities and separates native syntax from behavior without weakening scope, approval or execution restrictions. Quality inputs quote selected authoritative references and local help without exposing expected answers. Default evaluation remains mock-only; quality execution requires explicit account, model, disclosure and trial-budget approval.
  • Enumerate only legitimate task-scoped prerequisite/dependency reads. Preserve exact account/resource scopes, both independent post-create Foundry verification reads, alias mutation order, selected endpoint/authentication and rejection of sibling writes. Exact bootstrap Foundry prerequisite reads have regression cases rejecting foreign subscription/RG/account, omitted subscription, deployment creation, missing gateway creation and moving pending import into the approved plan. The exact scoped same-name gateway-family read is now accepted for alias name-selected operations, with foreign scope/name, omitted name, mutation and ordering negatives.
  • Separate approval operations from explanations using typed action objects: operation, identity type, role, gateway principal and exact scope. Assignment/removal vocabulary is neutral; required values come from the skill/evidence, not scenario answers. Reject removal/negated operations, broader roles, foreign principals/scopes, extra properties, missing approvals and moving pending work into approved commands. Preserve bounded legacy string compatibility for immutable historical outputs. The final original-budget trial demonstrated fresh passes for both previously failing approval scenarios. The subsequent case-only identity correction does not case-normalize CLI arguments, operation names, roles or resource IDs. The bootstrap input now explicitly selects single-deployment scope rather than broadening the oracle to accept account-wide import; the approved bulk scenario remains separate.
  • Add the canonical one-shot evals/waza/run_planning.sh. The actual launcher is inert-tested for empty/occupied results, missing identity/Python, changed source, unapproved model and saved failing benchmarks. It requires a prepared identity/configuration snapshot before Waza, keeps benchmark and policy-check statuses separately and verifies saved results even when the benchmark exits nonzero. It has no retries, authentication flow or orchestration loop.

Management entry size decreases from 13,445 to 9,752 bytes (149 lines); consumer entry size decreases from 33,131 to 6,767 bytes (101 lines). These are context-size measurements, not measured latency or quality gains. Detailed runtime guidance is retained in the reference. Evaluation must check whether progressive loading improves efficiency without causing important instructions to be missed. The new fixed-input planning profile does not measure progressive loading.

What is needed for an A

The target is reliable completion of major documented CLI scenarios and real gateway data-plane use, with performance reasonably close to equivalent CLI work. A passing synthetic planning suite alone cannot establish that.

Before assigning an A, require repeated agent scenario execution with all behavior and approval boundaries passing, observed model and MCP integration, relevant policy and operational scenarios, and matched CLI tool-count and latency evidence. Review task completion, CLI/contract accuracy, least-privilege selection, approval/recovery handling, and clear uncertainty reporting. The proposed rubric and paired-baseline budgets are in the evaluation guide; they remain criteria to test, not achieved results. Genuine held-out evaluation is absent; the three repeatedly inspected variants provide development evidence only.

Waza setup and next runs

Run from this PR's worktree, using a verified isolated profile and new processes. The user authorized assistant-owned execution of one corrected 18-task trial plus up to six additional one-trial invocations, all GPT-6.1 Sol. That allowance was used. The user then authorized up to three additional frozen trials; one was used before the bootstrap scope/identity correction. A subsequent explicit approval allowed three corrected frozen trials using the two remaining invocations plus one additional. Two were executed, the first 21/21 and the next 19/21. One invocation is unused. Globally, the initial trial plus nine additional invocations have been used. Paid execution stopped on the latest comparison failures and separate alias safety-preflight gap; one remaining invocation cannot establish the three-green target. No live Azure/data-plane calls or global plugin changes were authorized or made. Correct and test the demonstrated issues offline before seeking sufficient unchanged-candidate validation; do not spend remaining budget merely to chase a green result.

The operator's reported WorkIQ-only configuration did not establish the embedded runtime's startup boundary. That blocker was resolved through supported session-local COPILOT_HOME isolation, without modifying global installations. Actual non-model SDK stdio probes of embedded Copilot 1.0.105 confirmed explicit disableAllHooks: true, zero installed plugins, zero user MCP servers, no device-managed startup keys and existing authentication. The planning agent separately denies all model tools. This is not an OS filesystem, credential or network sandbox and does not make account policy immutable.

Use uv run --no-project --with pyyaml --with jsonschema for Python dependencies and grader subprocesses. Waza itself is the standalone Go binary, not a Python package. Installed Waza 0.40.1 was verified locally after initial PR publication. Disable automatic updates for reproducibility; do not treat environment flags as permission enforcement.

Offline checks and mock harness

The offline validation uses the existing Python environment:

PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s evals/waza -p 'test_*.py' -v

The operator commands use uv. Supplied smoke and quality artifacts predate the corrected prompts and rubric. Read-only replay preserves the original result and reports historical rubric revisions separately:

export PATH="$HOME/.local/bin:$PATH"
export WAZA_NO_UPDATE_CHECK=1
export COPILOT_AUTO_UPDATE=false
waza --version

uv run --no-project --with pyyaml --with jsonschema \
  python -m unittest discover -s evals/waza -p 'test_*.py' -q
uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/replay.py evals/waza/results/quality-gpt-6.1-sol.json

If the operator separately chooses another mock plumbing check, use a new filename rather than overwriting the supplied smoke artifact:

uv run --no-project --with pyyaml --with jsonschema \
  waza preflight evals/waza/eval.yaml --format json
# Stop on errors; review unresolved post-run boundary warnings before choosing a mock run.
mkdir -p evals/waza/results evals/waza/workspaces
SMOKE="evals/waza/results/smoke-$(date -u +%Y%m%dT%H%M%SZ).json"
test ! -e "$SMOKE" && TMPDIR="$PWD/evals/waza/workspaces" \
  uv run --no-project --with pyyaml --with jsonschema \
  waza run evals/waza/eval.yaml --no-cache --output "$SMOKE"

Stop on failed validation or preflight. Mock execution checks harness wiring and synthetic assertions, not assistant quality, secret handling, or cloud safety. The planning-only JSON checker below does not accept the mock profile. Strict preflight fails unresolved post-run boundary checks even for mock fixtures; a mock operator must review the non-strict report rather than treat it as a runtime assurance.

Real-model evaluation

The supplied gpt-6.1-sol / Copilot SDK artifact contains one trial per scenario (18 trials), with 0 original passes, 18 contract failures and no harness errors. The previous corrected-rubric replay remains 4/18; it is not a new trial of revised instructions. Review of the remaining 14 first failures found ten typed representation mismatches, one endpoint-versus-callable selector mismatch, one flattened approval-pending optional plan, and two arbitrary evaluator marker omissions. That classification does not certify every other field or establish unsafe execution. No narrative string is coerced into a successful boolean or execution approval.

The historical offline component assessment retains original evidence, including the old runtime URL: behavior has 7 pass, 2 contract failures and 9 unscored; representation/native syntax has 7 pass, 0 fail and 11 unscored. The two behavior-contract failures concern the bootstrap's flat conditional plan and the approved Foundry plan's additional reads/unmatched ordered command representation, not established unauthorized executions. Unknown facts and unrecorded native syntax remain unscored. Detailed REDACTED findings, field-only replay diagnostics and reported usage are in the journal linked below. No runtime skill rewrite was justified by those baseline outputs.

Keep gpt-6.1-sol fixed with one trial per task per invocation. The planning target is all 18 originals plus three development variants passing with complete policy evidence across three frozen trials; a change resets the repeatability streak. That target is unmet. The bounded allowance is a ceiling, not a quota. No second model, Claude model, automated workflow or unbounded rerun loop is used.

Regenerate configurations made before the evidence-resource and task-context corrections. Each task now supplies one legitimate evidence.json resource identical to inline synthetic evidence; selected recipe sections come directly from the runtime skill references, not oracle answers. The prior inline-only profile failed the pinned empty-sandbox relative-path guard. The corrected profile passed exact schema/binding and narrowly checked non-strict preflight before execution. The operator's earlier strict attempt stopped before any model trial, but preflight alone never verifies runtime enforcement. Configurations predating the latest bootstrap scope correction also require regeneration; the updated combined config was statically verified and executed in the two corrected frozen trials.

The original 18-task preparation commands are:

set -euo pipefail
uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/author.py \
  --quality --authorized --model gpt-6.1-sol --trials 1
uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml
waza --version
# Stop unless this is the approved v0.40.1 binary/runtime.
uv run --no-project --with pyyaml --with jsonschema \
  waza preflight evals/waza/authorized-quality/eval.yaml --format json | \
  uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --preflight -

Do not use --strict for this planning suite: pinned preflight never starts a runtime or program grader. Default exits 1 for invalid configuration; strict additionally fails all unresolved/unsupported diagnostics. There is no per-warning waiver flag. The JSON checker permits only boundary.requested_policy, grader.subprocess, boundary.post_run and runtime.availability as unresolved warnings, while requiring exact complete enabled-task inventory and the selected requested-deny-all policy/expected grader dependencies. It rejects invalid, unsupported or unexpected diagnostics/dependencies and missing warning inventory. A successful check means reviewed offline prerequisites, not runtime readiness or verified enforcement. Stop on any checker failure, then make the operator's runtime/account/environment and cost decision separately.

The supported planning boundary is configured: planning.agent.md declares tools: []; generated skill: aigateway-planning, skill_directories: [".."] and inject_skill_body: true bind it through Waza v0.40.1's agent resolution. Do not disable skills or substitute a direct filename: that can bypass policy resolution. The pinned executor applies an empty SDK tool filter, fail-closed permission callback and pre-tool-use denial hook. All model-initiated WorkIQ/MCP/action, shell, network and file/grader-answer reads are denied. Selected skill text and synthetic evidence are supplied inline, with no grader answers. The generated verifier rejects stale prompts, extra tasks and shadowed/modified policy binding.

The first corrected result records all 18 runs as deny_all with zero calls/events/denials; post-result verification passed. The later log-only run has no retained per-task policy records and cannot inherit that verification. Eight subsequent complete combined results retain complete 21-task deny-all evidence and zero activity. The supported SDK startup probe checks the actual isolated profile, unlike standalone CLI inventory. Waza's pinned stdio startup passes the process environment, so retain COPILOT_HOME on execution. This still does not confine the process filesystem, credentials, caches or provider transport. Stop for unexpected login, installation or startup policy changes; no personal configuration or credentials are copied into fixtures or logs.

Combined selection uses author.py --quality --authorized --model gpt-6.1-sol --trials 1 --include-holdouts, with authorized-combined/eval.yaml and the same binding/preflight checks. It preserves the original 18 and separately authored three-task selection. Across the final original-budget trial and first newly authorized frozen trial all 23 generated files, including 21 model inputs, were byte-identical. Grader corrections were not exposed as oracle answers or scenario recipes. After the offline scope correction, only the bootstrap task differs from that snapshot; all-task tests bind exact request/evidence and oracle subscription/RG/gateway/Foundry/deployment targets. The two corrected frozen trials retained the same skill, grader, fixtures and generated inputs throughout both executions.

Paid execution is currently stopped. If a future trial is explicitly resumed, use the tracked, inert-tested evals/waza/run_planning.sh, not ad hoc shell reconstruction. Before it accepts a fresh result path, prepare its exclusive .identity.json and .configuration/ snapshot, frozen source/configuration and immutable-prior hashes, approved model/trials and actual isolated-profile evidence. The recorded invocation takes this form, with a fresh, prepared result path rather than a previously saved artifact:

PLANNING_HOME="$PWD/evals/waza/workspaces/planning-home" \
  bash evals/waza/run_planning.sh \
  evals/waza/authorized-combined/eval.yaml \
  "$PREPARED_FRESH_RESULT"

The launcher refuses an empty/occupied output, missing receipt, changed inputs or unapproved model before Waza. It always checks an existing saved result after benchmark failure, captures both exit statuses, verifies source/prior immutability and retains result SHA-256. A benchmark failure is not hidden by successful policy verification. No retry is automatic.

Never overwrite supplied or prior result artifacts. My historical unchecked system python preparation failed, but the shell continued with an empty --output; v0.40.1 ran the benchmark without saving a full result. The ignored aggregate log is preserved, not fabricated into a result. Bare system python is unavailable; preparation now uses uv-managed Python. Post-run verification requires recorded deny_all on every run and zero tool calls/events/denials; missing evidence fails. Review blocked attempts even when prevented. The fixed-input profile does not measure plugin discovery, progressive loading, live gateway behavior or CLI-relative performance. No A claim is established.

Three separate development variants vary the Messages/Api-Key contract, selected-deployment identity/role approval and changed-source add-only sync with denied discovery. They were separately authored and statically validated, then included in eight complete combined executions. They have now been repeatedly inspected during development and are not genuine held-out evaluation. Historical holdout filenames, IDs and flags remain for compatibility. A baseline replay cannot validate them. A separate three-task configuration remains available through author.py --quality --authorized --model gpt-6.1-sol --trials 1 --holdouts and authorized-holdouts/eval.yaml; selection itself never authorizes another invocation or expands the ceiling.

Validation evidence

Evidence Result
Packaging, paths, context budgets, and bundled Python client example 10 tests passed; client requests simulated locally, no provider call
Migration safety regressions Previously recorded 135 tests passed from tests/migrate; deterministic helper implementation unchanged
Feedback regressions Previously recorded 9 tests passed from tests/feedback
Waza fixture/planning/schema/oracle/negative tests 57 tests passed across 18 baseline and three development variants, including pinned path-guard parity, context/oracle isolation, scope/approval/runtime negatives, bounded waits, typed approval safety, case-only identity normalization, all-task target binding and actual-launcher inert checks; rerun before the two corrected frozen trials
Portable root manifest Validated against the published Agent Plugins 1.0.0 schema
Local Copilot discovery Previous plugin aigateway 4.1.0, four skills and six adapters loaded using --plugin-dir with isolated COPILOT_HOME; no global installation changed by that check; not rerun in these correction passes
Whitespace and publication preflight git diff --check passed; remote source and follow-up refs verified
Subsequent operator setup Waza binary reports 0.40.1; uv and standalone Copilot available; operator's WorkIQ-only report was not sufficient startup evidence, replaced by actual isolated SDK checks
Supplied Waza mock harness artifact Reports 18/18 passes, no errors; synthetic assertions only
Supplied real-model artifact Copilot SDK / GPT-6.1 Sol, one run for all 18 tasks; 0 original passes, 18 contract failures, no harness errors
Historical corrected-rubric replay Previous 4/18 unchanged outputs pass; no new model execution
Historical offline component assessment Behavior 7 pass / 2 contract failures / 9 unscored; syntax 7 pass / 0 fail / 11 unscored; original input/artifact unchanged
Planning tool-policy binding Static binding and actual isolated SDK startup verified; all fully retained assistant-owned results have complete deny-all/zero-activity evidence; console-only run lacks per-task records; not an OS sandbox
First corrected real planning run 8/18 pass, 10 fail; benchmark exit 1, policy verifier exit 0; 30.9943 AI credits, 18 premium requests, 156.181 seconds
Corrected-result offline replay Behavior remains 8/18; syntax 16 pass/two unscored after recorded local-help coverage; immutable old output, not a fresh trial
Additional invocation 1 Console-only 12/18 pass, six behavior failures, aggregate errors 0; 35.79605 AI credits, 18 premium requests, 163.499 seconds; full result absent after shell preparation error; execution stopped until fail-closed preparation was corrected
Additional invocation 2 Fresh 11/18 originals + 2/3 variants = 13/21; complete policy evidence; 41.19985 credits, 21 premium requests, 179.355 seconds
Additional invocation 3 Fresh 15/18 originals + 2/3 variants = 17/21; complete policy evidence; 41.65935 credits, 21 premium requests, 180.577 seconds
Additional invocation 4 Fresh 17/18 originals + 3/3 variants = 20/21; complete policy evidence; 42.0811 credits, 21 premium requests, 192.570 seconds; separate unchanged-output replay after exact prerequisite-read correction passes 21/21 behavior and syntax
Additional invocation 5 Fresh 17/18 originals + 2/3 variants = 19/21; syntax 21/21; two approval-sentence comparisons remain; complete policy evidence; 42.0101 credits, 21 premium requests, 188.135 seconds; benchmark exit 1, policy verifier exit 0
Additional invocation 6 Fresh 17/18 originals + 3/3 variants = 20/21; syntax 21/21; typed approval cases passed; exact alias family read missing from grader, corrected offline with strict negatives; separate replay 21/21, fresh score unchanged; 42.65535 credits, 21 premium requests, 189.511 seconds; benchmark 1, policy 0
Additional invocation 7: first newly authorized frozen trial Fresh 17/18 originals + 3/3 variants = 20/21; syntax 21/21; bootstrap identity representation/import-scope comparison failed; no setup failures; complete policy evidence; 42.78910 credits, 21 premium requests, 181.589 seconds; benchmark 1, policy 0
Bootstrap offline correction/replay Request explicitly single-deployment; exact identity enum case normalized without broader operation/scope acceptance; only one generated task changed; replay retains old request, 20/21 behavior and 21/21 syntax, broader pending import still fails; this historical replay is not fresh corrected-input evidence
Additional invocation 8: first corrected frozen trial Fresh 18/18 originals + 3/3 variants = 21/21; syntax 21/21; bootstrap passed; all tasks reached inference; complete deny-all evidence, zero activity; 42.66460 credits, 21 premium requests, 187.779 seconds; benchmark 0, policy 0
Additional invocation 9: second unchanged corrected trial Fresh 16/18 originals + 3/3 variants = 19/21; syntax 21/21; bootstrap passed; alias scoped-list and composite-label comparisons failed; separate alias family-preflight omission/coverage gap identified; complete policy evidence; 42.50285 credits, 21 premium requests, 187.171 seconds; benchmark 1, policy 0; no post-run rubric change or score relabeling
Assistant-owned usage/budget Initial plus nine additional invocations used; latest three-trial approval used two invocations, one unused; 404.35265 AI credits, 204 premium requests/turns and 1,806.367 seconds reported suite duration, including console-only usage; paid execution stopped
Operator strict-preflight attempt Generation/binding check succeeded; strict rejected inherent runtime-only unknowns; && prevented a new model trial
Live gateway and CLI-relative performance Not measured

Live routing, protocol interoperability, policy enforcement, network readiness, and telemetry ingestion remain unverified. See evaluation guidance for authored versus executed evidence, authorization, isolation, and matched-baseline requirements.

Open questions

  • Bootstrap passed both fresh corrected trials; can an unchanged candidate pass every scenario across three fresh trials? The next trial after the first green result failed, so repeatability is not established. One remaining invocation cannot establish the target after a correction.
  • How should the eval accept exact scoped alias lists and equivalent composite-counter representation while independently requiring the same-name family precaution before name-selected writes? The latest plan omits that precaution, and comparison corrections alone cannot certify full adherence. Resolve and negative-test these issues offline before more spending.
  • Do repeated model trials complete the scenarios correctly without losing details moved into references? Fixed-input planning cannot answer progressive-loading questions.
  • Do observed gateway data-plane and operational scenarios, plus matched CLI baselines, meet the proposed A criteria? Those executions require separate authorization. Genuine held-out validation is also absent.

References and recorded Git provenance

Native-stack registration has not been performed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: The first run failed every contract check, but the eval rejected valid forms of some answers and hid required field types. This was not a shared startup failure or proof that all 18 skills scenarios were broken. Correct the eval and duplicate routing instructions, then replay the saved answers before considering another paid run.

Waza journal: 2026-11-09

The supplied local quality artifact records Copilot SDK / gpt-6.1-sol, 18 selected tasks, one run per task. Result: 0 passed, 18 failed contract checks, 0 harness errors, 0 skipped. All final outputs parse as JSON, all program graders executed, and offline replay reproduces the failures. This is not a shared startup, missing-file, dependency, or invalid-JSON failure, and 18 failed contracts do not establish 18 broken skills.

The rubric is too rigid in several high-confidence ways: it rejects additional relevant safety gates, full resource IDs where a short selected name is expected, and reasonable read-only/assessment decisions. Prompts name fields but do not specify their nested value types, so explanations are being compared with hidden booleans/enums. The alias URL oracle also omits the required /models segment. These are evaluation-contract defects to fix before drawing behavior conclusions. Two outputs omit required gate identifiers; these remain contract findings, not proof that the corresponding unsafe action occurred.

Reported wall duration is 288.669 seconds; median task duration is 27.804 seconds. Usage reports 74 turns, 1,658,977 input tokens, 16,668 output tokens, 18 premium requests, and 91.02379 AI credits. These are artifact-reported values, not independently billed charges. No matched CLI baseline, multi-trial reliability result, or Grade A is established. Recorded tool names are skill, glob, view and rg; no oracle-file reads were found in recorded arguments. That does not establish complete execution safety. The Waza v0.40.1 executor's permission defaults and this run's lack of an established enforced tool boundary prevent a safety certification.

Next: reduce duplicate adapters while preserving public skill names/boundaries, correct the high-confidence rubric defects, add typed output-shape guidance without exposing scenario answers, and replay these existing outputs offline. Any new paid trial requires separate authorization and an enforced execution boundary. No additional model trial, Azure operation, or data-plane call has been run for this review. Raw prompts, transcripts, local paths and credentials remain uncommitted and unpublished.

@paulyuk

Copy link
Copy Markdown
Member Author

In plain English, the first attempt to define and run a Waza eval was too rigid. but this helped us learn. doing an iteration now. we repeat in a loop to improve the evals, improve the skills, repeat.

Keep public skill boundaries and migration helpers unchanged, move development regressions outside runtime skills, and correct reviewed rubric defects without relaxing approval or execution gates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: Repeated adapter instructions and several overly rigid eval comparisons are corrected. Replaying the same answers now passes 4 of 18 scenarios; that is an offline comparison, not a new model score. Next, clarify the remaining answer types and approval-pending plans, and establish the tool boundary before another paid trial.

2026-11-09 UTC: routing and evaluation corrections

The four public skill names and boundaries are unchanged. Migration remains bundled; feedback is a support utility. The six compatibility commands now route to one authoritative skill recipe instead of repeating workflows across Markdown and Gemini adapters. Evidence-sufficient explanations skip management setup/discovery. Migration and feedback development regressions moved outside runtime skill directories; executable helpers, catalogs and schemas are unchanged. The portable manifest has no standard exclusion field, so this does not claim every host omits development files when downloading the repository.

Corrected the reviewed rubric defects: task prompts specify types without revealing expected answers; extra known gates and exact scoped full IDs are accepted without waiving required gates; reviewed assessment phases are accepted; alias URLs include /default/models; system identity is named precisely; harmless JSON output on create/update is accepted; readiness plans read current state before waiting and allow a shorter positive bounded timeout. Unrequested/foreign commands, changed routing values, unknown/duplicate gates, missing required gates and execution remain rejected.

Evidence Result
Supplied GPT-6.1 Sol / Copilot SDK trial Original result unchanged: 0/18 passes, 18 contract failures, no harness errors; one run for every task
Offline replay of those unchanged outputs 4/18 pass under corrected rubric: alias sibling preservation, exact alias endpoint, LRO reconciliation, secret/marker refusal; 14 retain contract failures. This is not another model trial
Deterministic checks 27 Waza schema/fixture/oracle/negative checks, 10 packaging/client-example checks, 135 migration regressions and 9 feedback regressions pass
Local loading Isolated COPILOT_HOME plus --plugin-dir discovers four skills and six adapters; no global installation changed and no model invoked
Supplied mock artifact Reports 18/18 passes, no errors; synthetic harness assertions only, not model behavior or safety

Remaining first-failure diagnostics mostly concern typed fields answered with prose or structured explanations. Runtime-only and stale-sync also omit required gate identifiers. The bootstrap proposal includes optional Foundry work awaiting separate approval, which the flat allowlist cannot represent as conditional. These findings do not prove the proposed commands executed. A first mismatch can hide further contract failures; no narrative string was coerced to a successful boolean, and no missing gate was waived.

Next: establish and verify the execution boundary before separately approving a new trial. The saved quality configuration still has the old untyped prompts; regenerate it before any rerun, then review/preflight the generated suite. The exact preparation and waza run commands and pinned v0.40.1 schema/executor provenance are in evals/waza/README.md and sources.json. Waza's default allow-all callback, disabled plugins, WorkIQ availability, a fresh shell, uv and temporary directories do not establish enforced isolation. No additional paid trial, Azure/data-plane call or A-grade claim was made. Matched CLI performance, repeated reliability and live gateway behavior remain unmeasured. Raw artifacts remain local and ignored.

Recorded base: 7874ce7f7c0f0674402d55b7de7eefacf8c70562.
Pushed head: 4e3e7db97f37e1e52effe5b25e739f867672ad13.
#32 and native-stack metadata remain untouched.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: Most remaining failures concern answer representation or plans that include work awaiting approval, not demonstrated unsafe execution. The planning contract now separates pending work from authorized commands, with negative safety tests retained. Regenerate the inputs and review the deny-all boundary before an operator-run trial; the replay is not a fresh score.

2026-11-09: REDACTED planning-contract corrections

The corrections are committed and pushed. Regenerate the planning configuration
before running it; the old generated prompts must not be reused. The operator
owns every Waza execution. I ran only offline tests and immutable replay, with no
Waza command, model call, paid trial, Azure/data-plane call or global plugin change.

Findings

The supplied baseline records Copilot SDK / GPT-6.1 Sol (gpt-6.1-sol),
18 selected tasks, one trial each: 0 original passes, 18 contract failures,
0 harness errors
. All final outputs were JSON and the program graders ran.
This was not a shared executor-startup or JSON-envelope failure.

Review of the 14 failures remaining after the previous 4/18 replay:

First-failure category Count Tasks
Typed representation mismatch 10 APIM assessment; approved Foundry onboarding; bounded response shapes; conflicting auth; MCP discovery; native scoped discovery; policy denial; private connection; telemetry; policy counters
Known endpoint versus callable-name selector 1 Denied management read
Flattened approval-pending optional plan 1 Selected-RG bootstrap
Arbitrary evaluator marker omission 2 Runtime-only integration; changed-source sync

This classification does not certify every other field or establish 14 skill
defects. In particular, runtime-only planned no management calls and sync planned
a fresh preview, despite missing evaluator markers. The bootstrap explicitly
stopped optional import until its identity/role approvals.

Corrections

The active quality contract now checks typed facts, scoped operations and verified
arguments, without decision/gate labels. Lossless structured tool names, residuals
and deployment identities are accepted; narrative is not converted into booleans.
Planned fallback is separate from unknown historical traffic. Approved commands
and optional pendingCommands carry separate exact operation/resource approvals.
An approval-pending import cannot enter the approved list. Only enumerated scoped
supplemental reads are accepted; private-connection explanation can use supplied
evidence without another read. Native syntax and behavior are separate checks.
Unrecorded routes/options are unverified, not automatically invalid CLI syntax.
No demonstrated behavior gap justified changing runtime skill recipes.

The generated suite binds Waza v0.40.1's supported tools: [] planning agent:
skill: aigateway-planning, skill_directories: [".."], inject_skill_body: true.
The pinned implementation applies an empty SDK tool filter, fail-closed permission
callback and pre-tool-use denial hook. All model tools, including WorkIQ/MCP,
shell/network and file/grader-answer reads, are denied. Selected skill text and
synthetic evidence are inline instead.

This binding is statically checked, not locally runtime-verified. It is not an
OS filesystem, credential or network sandbox, and does not cover ambient
SDK/CLI configuration reads, hooks/server startup, caches or provider transport.
Disabled plugins, uv, fresh shells and temporary/home directories do not establish
those boundaries. Use only the approved runtime/account/environment and stop for
unexpected login, installation or startup behavior. Post-run verification requires
recorded deny_all and zero tool calls/events/denials; missing evidence fails.
The new profile evaluates fixed-input planning, not plugin discovery, progressive
loading, live operations or CLI-relative latency.

Three separate holdouts vary the Messages/Api-Key endpoint contract, an unapproved
single-deployment identity/role change, and changed-source add-only sync with denied
discovery. They are authored and statically validated, not executed or validated
by replay of the baseline, and are excluded from the default 18-task run.

Offline evidence

Evidence Result
Original rubric replay 0/18, unchanged
Previous corrected-rubric replay 4/18, unchanged
New behavior assessment of old outputs with original evidence 7 pass, 2 contract failures, 9 unscored
Separate representation/native-syntax assessment 7 pass, 0 fail, 11 unscored
Offline Waza fixture/planning/schema/negative tests 36 passed
Packaging tests 10 passed
Migration / feedback regressions 135 / 9 passed; runtime helpers unchanged
Local plugin loading Previous isolated four-skill/six-adapter check only; not rerun here
Fresh Waza/model execution None

The two remaining behavior-contract failures are the flat conditional bootstrap
plan and the approved Foundry plan's additional reads/unmatched ordered command
representation. Neither establishes an unauthorized executed action. Remaining
typed facts/unverified syntax need a newly authorized operator run or review.
replay.py reports each field-only diagnostic and the historical rubric revisions;
it never overwrites the input. No fresh score or A grade is claimed.

The original artifact SHA-256 remains
bacc2ce746c1f5902dda264bb39e8b931b4d2a25bcf1beb187a771ed22c4d730.
Its reported duration was 288,669 ms (median task 27,803.5 ms); reported usage was
74 turns, 1,658,977 input tokens, 16,668 output tokens, 1,418,239 cache-read tokens,
240,516 cache-write tokens, 18 premium requests and 91.02379 AI credits.
Those are artifact values, not independently verified charges or a matched CLI
baseline. Raw results, prompts, transcripts and replay artifacts remain ignored.

Operator terminal commands

From this PR worktree, stop on any failed check:

uv run --no-project --with pyyaml --with jsonschema \
  python -m unittest discover -s evals/waza -p 'test_*.py' -q
uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/replay.py evals/waza/results/quality-gpt-6.1-sol.json
uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/author.py --quality --authorized --model gpt-6.1-sol --trials 1
uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml
waza --version
# Stop unless this is the approved v0.40.1 binary/runtime.
uv run --no-project --with pyyaml --with jsonschema \
  waza preflight evals/waza/authorized-quality/eval.yaml --strict

After the operator's model-cost and environment decision, one new run:

mkdir -p evals/waza/results evals/waza/workspaces
RESULT="evals/waza/results/quality-planning-gpt-6.1-sol-$(date -u +%Y%m%dT%H%M%SZ).json"
test ! -e "$RESULT" && \
  TMPDIR="$PWD/evals/waza/workspaces" COPILOT_AUTO_UPDATE=false WAZA_NO_UPDATE_CHECK=1 \
  uv run --no-project --with pyyaml --with jsonschema \
  waza run evals/waza/authorized-quality/eval.yaml --no-cache --output "$RESULT" && \
  uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --result "$RESULT"

Never overwrite quality-gpt-6.1-sol.json. No retry/hill-climbing loop or additional
trial is authorized or started by this correction.

Provenance

  • Waza tag v0.40.1, commit e0c251df896bee4e0f9b38a693a6105bf7eadd19:
    policy,
    binding,
    agent parser
    and resolution.
    Exact schema digests, commands and limits are in evals/waza/sources.json
    and evals/waza/README.md.
  • Base/source SHA: 7874ce7f7c0f0674402d55b7de7eefacf8c70562.
  • Pushed head SHA: 6bbf7eaec007aaa99d79ed217bc528e902ec6e26.
  • Owning branch: paulyuk-ai-gateway-skill-improvements;
    base: feature/apim-migration-native-sync-20261105.
  • Original Update plugin to latest CLI and RP #32 and native-stack metadata remain untouched.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: Our strict-preflight instruction guaranteed failure on checks that cannot be verified offline. The operator's chain correctly stopped before spending model tokens. Use the narrowly checked non-strict report instead, while keeping actual configuration errors fatal and deciding separately whether runtime and cost are acceptable.

2026-11-09: Correct the offline preflight gate

The earlier --strict instruction was wrong for this suite. The operator's
configuration generation and binding check succeeded (deny_all, GPT-6.1 Sol,
18 tasks, one trial each). Strict preflight then failed on deliberately unresolved
offline diagnostics; the && chain correctly stopped before a model run. No new
trial occurred from that chain.

Pinned Waza v0.40.1 preflight never starts an engine, model, grader subprocess or
live service. Default preflight exits 1 for invalid configuration. --strict
additionally exits 1 for every unresolved or unsupported diagnostic, including
these unavoidable checks:

Diagnostic Why offline preflight cannot verify it
boundary.requested_policy Agent frontmatter records requested policy, not effective runtime enforcement
grader.subprocess Program graders are inspected but not executed
boundary.post_run Boundary graders need recorded post-run evidence
runtime.availability Runtime, model, authentication, permissions and effects are not checked offline

There is no supported per-warning waiver option. The correction uses supported
non-strict --format json output and a narrow report checker instead. It accepts
only those four unresolved warning codes, and still requires exact complete
enabled-task inventory, the selected requested-deny-all policy and expected
grader dependencies. Invalid configuration, unsupported/unknown diagnostics,
unexpected dependencies, missing policy evidence or incomplete inventory fail.
Generated-configuration verification still runs. This is not a blanket warning
bypass and does not mark runtime policy verified.

Exact next command, from this PR worktree after the pushed correction:

uv run --no-project --with pyyaml --with jsonschema \
  waza preflight evals/waza/authorized-quality/eval.yaml --format json | \
  uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --preflight -

Stop if the checker exits nonzero. The successful generated configuration need
not be regenerated: this correction changes preflight verification/docs, not
task prompts. A successful report means reviewed offline prerequisites only.
The operator still decides whether the approved runtime/account/environment and
cost are acceptable; the policy is not an OS filesystem/credential/network sandbox
and does not cover ambient hooks, server startup or provider transport.

After that decision, the previously published new-filename run stays unchanged:

mkdir -p evals/waza/results evals/waza/workspaces
RESULT="evals/waza/results/quality-planning-gpt-6.1-sol-$(date -u +%Y%m%dT%H%M%SZ).json"
test ! -e "$RESULT" && \
  TMPDIR="$PWD/evals/waza/workspaces" COPILOT_AUTO_UPDATE=false WAZA_NO_UPDATE_CHECK=1 \
  uv run --no-project --with pyyaml --with jsonschema \
  waza run evals/waza/authorized-quality/eval.yaml --no-cache --output "$RESULT" && \
  uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --result "$RESULT"

Never overwrite the supplied baseline. Post-run evidence must still record
deny_all, zero calls/events/denials and available tool counts. Missing evidence
fails. No Waza, mock, model or Azure command was executed by the implementing
agent; no result artifact was changed. 37 offline tests passed, including
synthetic known-warning acceptance and invalid/unsupported/unknown/incomplete/
wrong-policy rejection. This does not establish live enforcement.

References

Publication note

The prepared journal text above is restored here because the original comment displayed only this local-file reference: @evals/waza/results/pr38-preflight-correction-journal.md. Its commands are retained as historical instructions, not the current launch recipe.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: The eval spent tokens but 17 scenarios never reached the model because their quoted relative paths had no workspace input. Each task now supplies its real synthetic evidence file, and saved failures still reach the policy verifier. Regenerate the configuration before another approved run; the recorded failures are not 18 skill regressions.

2026-11-09: Planning input wiring and saved-result verification

The operator's new result spent tokens, but 17 tasks failed before model
execution because the profile supplied no workspace files
. This was a shared
input-wiring defect, not 18 skill regressions. The correction is pushed; regenerate
the configuration before considering another separately authorized trial.

Verified REDACTED result

Source: immutable quality-planning-gpt-6.1-sol-20261109T031346Z.json.
Executor Copilot SDK, model GPT-6.1 Sol, one configured trial for each of 18 tasks.
The operator reports Waza v0.40.1 and embedded Copilot cache 1.0.105.

Evidence Result
Aggregate task summary 0 passed / 18 failed / 0 aggregate errors
Actual executor setup failures 17 runs contain the same relative-path/empty-sandbox error_msg; no final output or recorded policy
Task reaching the model apim-assess-not-execute: syntax passes; behavior fails observations.residuals
Behavior mismatch Three narrative residual entries instead of the compact supported residual facts; no demonstrated unauthorized execution
Recorded policy evidence One executed run records deny_all; 17 policies missing; zero recorded tool calls/events/denials
Offline policy-check exit 1: whole-suite evidence fails, not a permission violation inferred from missing records
Artifact-reported usage One turn, one premium request, 10,430 input tokens, 759 output tokens, 10,427 cache-write tokens, 0 cache-read tokens, 3.36635 AI credits
Artifact-reported elapsed time 25.2 seconds; not a CLI-relative latency measurement or verified currency charge

The aggregate errors: 0 is not proof of an error-free executor. The run's nonzero
status and original && chain prevented the post-run checker from appearing in
the pasted terminal output. The implementing agent ran that checker offline,
without executing Waza or a model. Original 0/18 and historical replay results
remain separate and unchanged; no new score is claimed for the correction.

Root and correction

Pinned Waza runner.go:1846-1857 rejects a task when loaded resources are empty
and its prompt contains ./ or ../ anywhere. Relative links inside the quoted
management/consumer/feedback skill text trigger it. Migration's truncated
assessment text did not trigger the guard, explaining the single actual model
request. The previous generator removed inputs.files, so those inputs were
schema-valid but not executable through this guard.

The generator now supplies one legitimate evidence.json inline resource per
task
, exactly matching the evidence already supplied in the prompt. Waza's
runner.go:2092-2098 loads nonempty inline content before the guard. No dummy
file, broad directory, grader source or oracle answer is copied. File tools remain
denied; relative skill links are context, not requests to load additional files.
No guard, approval requirement, rubric or runtime skill was weakened.

39 offline tests passed. The new tests reproduce the pinned guard and inline
loading predicate against all 18 baseline and three holdout generated prompts,
reject the former empty-resource shape, check real resource content and existing
schemas/oracle isolation, and exercise saved-result exit handling with inert
commands substituted before shell execution. No Waza, mock, model, Azure or
provider call was executed by the implementing agent.

Operator commands

Regeneration is required. From this PR worktree at the pushed head, prepare
and check the one-trial GPT-6.1 Sol profile. Stop on any failure:

export WAZA_NO_UPDATE_CHECK=1
export COPILOT_AUTO_UPDATE=false
uv run --no-project --with pyyaml --with jsonschema \
  python -m unittest discover -s evals/waza -p 'test_*.py' -q && \
uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/author.py --quality --authorized --model gpt-6.1-sol --trials 1 && \
uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml

Use the already approved Waza v0.40.1/runtime; stop on unexpected installation
or login. Review offline prerequisites, not strict runtime-only unknowns:

uv run --no-project --with pyyaml --with jsonschema \
  waza preflight evals/waza/authorized-quality/eval.yaml --format json | \
uv run --no-project --with pyyaml --with jsonschema \
  python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --preflight -

Only after a successful check and the operator's renewed environment/cost
decision, run to a new filename. This sequence verifies a saved result even when
Waza reports benchmark failure, without hiding either exit status:

mkdir -p evals/waza/results evals/waza/workspaces
RESULT="evals/waza/results/quality-planning-gpt-6.1-sol-$(date -u +%Y%m%dT%H%M%SZ).json"
if test -e "$RESULT"; then
  printf 'Refusing to overwrite %s\n' "$RESULT"
  false
else
  if TMPDIR="$PWD/evals/waza/workspaces" COPILOT_AUTO_UPDATE=false WAZA_NO_UPDATE_CHECK=1 \
    uv run --no-project --with pyyaml --with jsonschema \
    waza run evals/waza/authorized-quality/eval.yaml --no-cache --output "$RESULT"; then
    WAZA_STATUS=0
  else
    WAZA_STATUS=$?
  fi
  if test -f "$RESULT"; then
    if uv run --no-project --with pyyaml --with jsonschema \
      python evals/waza/verify_planning.py evals/waza/authorized-quality/eval.yaml --result "$RESULT"; then
      POLICY_STATUS=0
    else
      POLICY_STATUS=$?
    fi
  else
    printf 'No saved result to verify\n'
    POLICY_STATUS=1
  fi
  printf 'Result: %s\nWaza exit: %s\nPolicy-check exit: %s\n' "$RESULT" "$WAZA_STATUS" "$POLICY_STATUS"
  test "$WAZA_STATUS" -eq 0 && test "$POLICY_STATUS" -eq 0
fi

Whole-suite runtime enforcement and scenario reliability remain unresolved. The
one task's policy record is evidence for that task only, not an OS filesystem,
credential, network or ambient-startup sandbox. Preflight and fixed-input tests
do not prove progressive skill loading, plugin discovery or live gateway quality.
There is no automated rerun or hill-climbing loop.

References

  • Pinned input guard,
    resource loading
    and ordering.
    Source SHA-256 and offline test provenance are in evals/waza/sources.json.
  • New artifact SHA-256:
    7fedf25456f5f8c046698247cb8bb9c45a004e563bc2aa8ef870cdb1952b175f;
    original baseline SHA-256:
    bacc2ce746c1f5902dda264bb39e8b931b4d2a25bcf1beb187a771ed22c4d730.
    Both checked unchanged after edits.
  • This corrects input wiring and the run/verification chain from the
    preflight journal.
  • Pushed head 7f62cdad30697199a3d290d77c4a2f78b386408f;
    base 7874ce7f7c0f0674402d55b7de7eefacf8c70562.
    Update plugin to latest CLI and RP #32 and native-stack metadata remain untouched.

Publication note

The prepared journal text above is restored here because the original comment displayed only this local-file reference: @evals/waza/results/pr38-input-resource-journal.md. Its commands are retained as historical instructions, not the current launch recipe.

@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: The input fix let all 18 scenarios run, with verified deny-all records, but only 8 passed. Several plans lacked the task recipes that the eval had omitted while disabling file tools; other failures need semantic review. Supply the missing authoritative context and review the comparisons offline before using the next approved trial.

2026-11-09: corrected planning run

The corrected input wiring completed all 18 original scenarios with Copilot SDK, GPT-6.1 Sol, one trial per task, Waza 0.40.1 and embedded runtime 1.0.105. Fresh result: 8 passed, 10 failed. All 18 recorded deny-all, zero tool calls/events and zero denials. No shared setup errors were found. Benchmark exit was 1; the post-result policy verifier still ran and passed with exit 0. This is planning evidence, not live Azure or an A grade.

Before the run, the actual embedded runtime was probed over its supported SDK stdio connection without creating a session or sending a prompt. An isolated COPILOT_HOME supplied disableAllHooks: true; the runtime recognized that explicit setting and reported zero installed plugins, zero user MCP servers and no device-managed hook/MCP/plugin keys. Existing authentication was available. No global plugin state or personal configuration was changed or copied. This does not provide an OS filesystem, credential or network sandbox.

Failed-task group Count Finding
Missing or incomplete native command plans 6 Alias update, approved Foundry import, bootstrap, native alias discovery, LRO reconciliation and stale sync. Several answers stopped at help/read prerequisites. The fixed-input generator supplied skill entry points but omitted their referenced task recipes while denying all file/skill tools. This is a demonstrated context omission to correct before spending another trial, not six established unsafe executions.
Runtime proposal versus native-only command representation 1 The runtime-only task supplied a curl proposal. The evaluator's native-command-only comparison rejected it. Request correctness and response bounds still need review; a non-native proposal must not automatically become a successful safety result.
Fact representation/comparison 3 APIM residuals were narrative entries; the denied second page was reported as a known lower bound with an unknown total; token/request/composite-counter units were described precisely rather than with the evaluator's categorical strings. These need semantic review without dropping unsupported-policy, unknown-total or quota-boundary requirements.

Native-syntax component: 13 pass, five unscored, zero established invalid-syntax findings. The five unscored outcomes are not passes. No skill source was changed during the trial. Source/configuration digests and the two supplied original artifacts remained unchanged; a separate configuration snapshot, identity record and both exit statuses are preserved locally.

Reported usage: 18 turns, 18 premium requests, 107,460 input tokens, 9,532 output tokens, 22,500 cache-read and 84,906 cache-write tokens, 30.9943 AI credits, and 156.181 seconds elapsed. These are artifact-reported usage and latency, not verified currency charges or matched CLI performance.

Next: correct the missing task-reference context and high-confidence evaluator representation defects offline, retain negative safety checks, then use the approved bounded budget only if those checks pass. This used the initial corrected invocation; six additional one-trial invocations remain available, not required. The three unseen holdouts and three-trial repeatability are not yet established.

References: local immutable result quality-planning-gpt-6.1-sol-20261109T032753Z.json; earlier input-wiring correction. Raw prompts, final outputs, traces and credentials are not published.

Accept task-scoped local help, safer exact gateway reads and bounded readiness waits without relaxing approval, mutation or runtime gates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Stop on preparation errors, require the isolated profile and a nonempty unique result path, and preserve benchmark versus policy-verifier statuses independently.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: The next run's console reported 12 of 18 passes, but our launch preparation lost the full result and policy evidence. That score cannot establish reliability, and paid execution is stopped. The launch path is now fail-fast and preserves both exit statuses; verify that exact path before resuming.

2026-11-09: isolated planning runs, context correction and stopped execution

Planning completion is not established. The first corrected run has complete
tool-policy evidence and passed 8/18. The next run's console summary improved
to 12/18, but my shell-preparation error prevented saving its full result and
running post-result verification. I stopped further paid invocations.

Actual execution and evidence

Both invocations used Waza v0.40.1, the Copilot SDK executor, embedded Copilot
1.0.105, GPT-6.1 Sol, the original 18 scenarios and one trial per task.
No holdout or repeatability trial was executed.

Evidence First corrected run Next run
Reported passes / failures 8 / 10 12 / 6, console aggregate only
Benchmark exit 1 1
Post-result policy verification Passed: all 18 deny-all; zero calls/events/denials Not executed; no full result saved
Reported premium requests / turns 18 / 18 18 / 18
Reported AI credits 30.9943 35.79605
Input / output tokens 107,460 / 9,532 120,411 / 11,096
Cache-read / cache-write tokens 22,500 / 84,906 22,500 / 97,857
Elapsed seconds 156.181 163.499

Combined reported usage is 66.79035 AI credits and 36 premium requests. These
are harness-reported units, not a verified currency charge or CLI-relative
performance measure. One initial and one additional invocation were used;
five of the six additional authorized invocations remain unused.

The actual embedded runtime was probed through its supported SDK stdio APIs
without creating a model session. Its isolated COPILOT_HOME reported explicit
disableAllHooks: true, zero installed plugins, zero user MCP servers, no
device-managed hook/MCP/plugin keys and already available authentication.
No global plugin configuration or personal credentials were copied or changed.
This verifies that startup profile, not an OS filesystem/credential/network
sandbox. The later run cannot inherit the first run's policy verification.

Corrections and failure classification

The first run's ten failures included six missing native operation plans where
the denied-tool profile supplied only the entry point, not its referenced task
recipes. Four other first-failure findings concerned runtime-example placement,
residual representation, uncertainty/count representation and policy unit
representation. These findings do not establish unsafe execution.

Quality inputs now quote selected sections directly from the authoritative
skill references and supply recorded local CLI help. Oracle answers, graders
and expected command lists remain excluded from agent inputs. Runtime skill
names, instructions and boundaries were not changed.

The grader now accepts recorded local version/help probes without letting them
replace required operations, an exact typed gateway-ID read in place of reviewed
name selectors, and shorter positive readiness bounds. It does not require
repeating an initial read already supplied in evidence. Wrong targets, missing
read-back, unapproved changes, runtime-only management probes and unsafe waits
remain rejected. Offline replay of the immutable first corrected result remains
8/18 for behavior; syntax is 16 pass/two unscored under the expanded recorded-help
coverage. This is not a new score for those old outputs.

The next run reported behavior failures for approved Foundry onboarding, exact
alias endpoint, native task-scoped discovery, selected-RG bootstrap, stale-sync
preview and token/request counters. Without retained per-task outputs, their
root causes cannot be classified responsibly.

Shell failure and next step

My preparation command used unavailable system python without fail-fast
shell handling. The shell continued and invoked Waza with an empty --output;
v0.40.1 completed the benchmark but saved no result. The ignored console log and
an explicitly aggregate-only summary are preserved. No result, policy record or
pre-run identity has been fabricated. The supplied artifacts and the first
corrected result are unchanged byte-for-byte.

The documented launch path now uses set -euo pipefail, requires the verified
isolated profile and a nonempty unique result path, and keeps benchmark and
policy-verifier statuses separate. Inert shell regression tests prove that
missing preparation/profile inputs cannot launch the paid command and that
a saved failing result still reaches the verifier. All 43 targeted offline
tests pass.

Further paid runs remain stopped because full execution evidence is missing,
not because the trial ceiling is exhausted. Planning still needs all original
18 plus three unexecuted holdouts passing across three frozen trials with no
unresolved execution/policy failures. Live Azure/data-plane checks and a matched
CLI performance baseline are separately unauthorized. No Grade A claim is made.

References: first corrected run journal;
evals/waza/README.md, author.py, grade.py, test_planning.py;
Waza v0.40.1 pinned schema/source provenance in evals/waza/sources.json.

Preserve all 21 independent inputs and grader selections, validate exact combined inventory and exclude generated configuration from source control.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: The complete fresh run passed 13 of 21 scenarios. Most failures were equivalent facts or relevant scoped reads rejected by the eval; the consumer skill also needed clearer guidance for an optional typed gateway read. Those corrections replay at 20 of 21, not a new model score. Next, regenerate and freeze the corrected inputs for the next approved trial.

Planning evaluation, 2026-11-09

The latest complete trial passed 13/21: 11/18 original scenarios and 2/3 holdouts. This is development evidence, not an A grade or repeatability result.

Executor: Waza 0.40.1, Copilot SDK with embedded runtime 1.0.105, GPT-6.1 Sol, one trial per task. Every task recorded requested deny_all, zero tool calls/events/denials and complete policy counts. Benchmark exit was 1; the independent post-result policy verifier exited 0. There were no identified setup/runtime-policy failures in this retained execution; aggregate errors: 0 alone was not used as proof.

The eight failed tasks divide into four fact/representation comparisons, three scoped-plan comparisons, and one consumer-guidance gap:

  • Full URL versus the exact published operation path, composite identity/IP spelling, SystemAssigned/approval wording and the holdout's account-role approval wording were semantically equivalent.
  • The alias, Foundry and stale-sync plans contained relevant scoped prerequisite/dependency reads. Foundry's independent model and identity verification reads were in the opposite order. Missing core Azure help coverage also left its syntax unscored.
  • Optional consumer gateway-root inspection used name selectors instead of the supplied typed ID. The skill now explicitly prefers show --ids for that read; root inspection is not mandatory.

The evaluator corrections accept only lossless, bounded equivalents and enumerated reads. They retain exact gateway/account scope, both identity and Foundry User approvals, composite rather than independent counters, create-before-verification, alias read/update/read ordering and rejection of sibling mutations. Local core Azure help, not Azure calls, established the additional read-option and colored-JSON contracts. The source change does not teach evaluator labels or task answers.

Offline replay of unchanged outputs: 20/21 behavior and 21/21 syntax pass. The remaining native-discovery failure is retained. This replay does not change the recorded fresh 13/21 score or test the skill correction. Regression checks include positive equivalents and negative cases for foreign scopes, broader roles, missing verification, wrong endpoints, table output and reordered/unrequested mutations.

Usage: 21 turns/premium requests; 140,596 input and 12,354 output tokens; cache-read 26,250 and cache-write 114,283 tokens; 41.19985 AI credits; 179.355 seconds. Including the earlier complete run and the honestly recorded log-only invocation, assistant-owned usage is 107.9902 credits and 57 premium requests. The initial invocation plus two of six additional invocations have been used; four additional remain.

The actual embedded runtime's session-local profile was checked without a model prompt: hooks disabled, no installed plugins or user MCP configuration, no device-managed startup keys, existing authentication. This and deny-all records are not an OS filesystem, credential or network sandbox. No global plugin configuration, live Azure/data-plane action or WorkIQ action was changed or invoked.

Next: regenerate and check the corrected combined profile, freeze the candidate, then use the next authorized one-trial invocation. Completion still requires all 21 with complete policy evidence across three frozen trials. Holdouts already inspected during development must not be described as newly unseen acceptance. All raw results, configuration snapshots and prior replays remain local and immutable; none are attached here.

Reference: retained combined result SHA256 8bdce752bc562968de81bd23f07aaac9573c1970190a9ac02c4a5d65e55503ab.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: The fresh run passed 17 of 21; the remaining failures were bounded wording differences and correctly scoped prerequisite reads, not identified setup or unsafe-action failures. Correcting those comparisons makes the saved-answer replay green, but does not change the fresh score. Next, run the frozen candidate through the recorded one-shot launcher and check consistency.

Planning corrections and retained trial, 2026-11-09

The next retained GPT-6.1 Sol trial passed 17/21: 15/18 originals and 2/3 holdouts. All 21 have complete deny-all evidence and zero calls/events/denials. Waza exited 1; the policy verifier exited 0. Source/configuration and all prior result artifacts were unchanged during execution.

Waza 0.40.1 used Copilot SDK/runtime 1.0.105, one trial per task. Usage was 21 premium requests/turns, 142,330 input and 12,380 output tokens, cache-read 26,250/cache-write 116,017, 41.65935 AI credits and 180.577 seconds. Total assistant-owned usage so far is 149.64955 credits and 78 premium requests. Initial plus three additional invocations are used; three additional remain under the unchanged ceiling.

The four failures were evaluator comparisons, not identified setup, policy or missing-skill behavior:

  • The holdout used precise natural approval sentences with punctuation and a more explicit system-identity/account-scope description.
  • Runtime-only integration marked unread backend/policy metadata unavailable rather than unknown, as the fixed-input instructions permit. It did not claim an empty policy set or runtime failure.
  • Bootstrap included the scoped same-name-family resource read required by the lifecycle guidance, used the equivalent system-assigned identity term and omitted “managed” in its identity approval sentence.
  • Private-connection review included the correctly typed gateway-root read before the selected connection read.

Corrections accept those bounded equivalents and exact scoped reads. Approval parsing is anchored to the intended identity/Foundry User operation and still requires both exact scopes; negation, user-assigned identity, added/broader roles and multiple action sentences fail. Missing-metadata equivalence is limited to the corresponding configuration fields, not private reachability. Foreign targets and unapproved mutations remain rejected.

Offline replay: 21/21 behavior and 21/21 syntax. That is a separate replay, not a changed fresh score or proof that a new trial is green.

Recorded execution now uses one canonical, one-shot run_planning.sh, not reconstructed shell commands. It requires an existing identity/snapshot receipt, fresh nonempty result, writable parent, unchanged inputs/originals and the approved model/profile before Waza. It always checks a saved failing result, records benchmark/verifier exits separately and checks post-run immutability. Inert tests exercise this actual file with empty/occupied output, missing Python/identity, changed source, wrong model and a saved failing benchmark. It performs no retries or authentication.

The corrected candidate will be frozen for the next authorized invocation. If green, the two remaining invocations can test unchanged repeatability. No live Azure/data-plane/WorkIQ actions or global plugin changes are authorized or performed. Startup-profile/tool-policy evidence is still not an OS sandbox. The inspected holdouts remain a distinct development cohort, not newly unseen acceptance evidence.

Reference: retained fresh-result SHA256 78caf739a40a6577ed18529cbba1b025f0ec2dc708f71eaa4823cb15419e56ea.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: The fresh run passed 20 of 21; bootstrap failed because the eval rejected necessary reads of the selected Foundry account and deployments. Those reads are now accepted without permitting foreign scopes or unapproved imports. The offline replay is green; next, check the unchanged inputs in another authorized fresh run, not treat replay as repeatability.

Planning evaluation journal: November 9, 2026

The latest fresh trial passed 17/18 original scenarios and 3/3 holdouts, or 20/21 overall. Waza 0.40.1 used the Copilot SDK executor, embedded runtime 1.0.105, GPT-6.1 Sol and one trial per task. All 21 tasks reached the model. Syntax passed throughout. The sole behavior failure was the bootstrap grader rejecting read-only checks of the exact selected Foundry account and its deployments, not an unauthorized import or role grant.

Those prerequisite reads are now explicitly enumerated for bootstrap. Regression cases reject a different subscription, RG or account, omitted subscription, deployment creation, missing gateway creation and moving pending Foundry import into the approved plan. No runtime skill or model input changed. The regenerated configuration's 23 files are byte-identical to this trial's configuration. Offline replay of its immutable outputs now passes 21/21 behavior and syntax checks; the recorded fresh score remains 20/21.

All 21 results recorded deny-all policy and zero tool calls, events or denials. The benchmark exited 1 and the independently executed policy verifier exited 0. Frozen source/configuration and earlier artifacts remained unchanged. Sessionless SDK checks confirmed hooks disabled, no installed plugins or user MCP servers, no managed startup policy keys and existing authentication. These controls are not an OS filesystem, credential or network sandbox.

This trial used 42.0811 AI credits, 21 premium requests/turns, 142,345 input tokens, 12,798 output tokens, 26,250 cache-read tokens and 116,032 cache-write tokens, over 192.570 seconds. Assistant-owned usage so far is 191.73065 credits and 99 premium requests/turns, including the previously disclosed console-only invocation.

The initial authorized trial and four of six additional invocations have been used. Two additional invocations remain. There is no fresh all-green repeatability streak yet. Even if both remaining trials pass, three unchanged-candidate green trials cannot be established within this ceiling. The next bounded trial can establish fresh coverage after the prerequisite comparison correction; no extra authorization is inferred.

The holdouts were independently authored but have been inspected during development. They are not newly blind acceptance evidence. No live Azure/data-plane calls, matched CLI performance baseline or Grade A claim is established.

Evidence retained locally: immutable result quality-combined-gpt-6.1-sol-20261109T042603Z.json, configuration snapshot, pre-run identity and independent exit-status receipt. Result SHA-256: 0618fbac01e773d4da9d38a672dbe9838ed88ddbf9c1ecb1f92dfe9b87de9fd5. No raw prompts, transcripts, credentials or workplace data are published.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: Two fresh failures came from valid pending approval operations expressed outside the eval's sentence grammar, not an unauthorized grant or import. Paid execution is stopped rather than adding more wording exceptions. Next, separate typed operations and exact scopes from explanatory prose, then validate that correction before another approved run.

Planning evaluation journal: November 9, 2026

Execution is stopped; planning completion is not established. The latest fresh trial passed 17/18 original scenarios and 2/3 holdouts, or 19/21. The preceding trial passed 20/21. No all-green fresh trial or three-trial repeatability streak exists.

The two remaining failures are approval-action representation comparisons in bootstrap and the Foundry account-role-scope holdout. Both outputs proposed the exact reviewed pending command, retained both exact approval scopes, reported no execution and made no tool calls. Their explanations distinguished an account-wide Foundry User grant from deployment-only scope using sentences outside the grader's anchored grammar. Native syntax passed all 21 tasks. No setup or policy failure was found. These observations do not convert the recorded failures into passes.

The exact Foundry prerequisite-read omission found in the preceding trial was corrected, with tests rejecting foreign subscriptions/RGs/accounts, omitted subscription, deployment creation, missing gateway creation and unapproved import. All 23 regenerated model input/configuration files remained byte-identical to the preceding trial. Offline replay of that earlier immutable output is 21/21 behavior and syntax; the latest fresh output remains 19/21 behavior and 21/21 syntax.

I stopped on repeated approval-sentence comparison failures instead of adding another wording special case and spending the remaining invocation. The unresolved design is to separate typed approval operations from explanatory prose while retaining exact command, identity, role and scope gates. No further rubric change or paid run has been made.

The latest invocation used Waza 0.40.1, Copilot SDK / embedded runtime 1.0.105, GPT-6.1 Sol, one trial per each of 21 tasks. It consumed 42.0101 AI credits, 21 premium requests/turns, 142,341 input tokens, 12,728 output tokens, 26,250 cache-read tokens and 116,028 cache-write tokens, over 188.135 seconds.

All 21 runs recorded deny-all with zero tool calls/events/denials and complete policy counts. Benchmark exit was 1; independent saved-result policy verification exited 0. Pre-run configuration identity and post-run hashes show source/configuration and earlier artifacts unchanged. The actual session-local SDK startup had hooks disabled, no installed plugins or user MCP servers, no managed startup policy keys and existing authentication. This is not an OS filesystem, credential or network sandbox.

Budget: the initial corrected invocation plus five of six additional invocations have been used. One additional invocation remains unused. Assistant-owned totals, including the disclosed console-only run, are 233.74075 AI credits, 120 premium requests/turns and 1,060.317 seconds of reported suite duration. This ceiling is not a quota.

The three holdouts were independently authored but inspected during development; they are not newly blind acceptance evidence. Live Azure/data-plane behavior, progressive plugin loading and matched CLI performance remain unmeasured and separately authorized. No Grade A claim is made.

Evidence: latest immutable result quality-combined-gpt-6.1-sol-20261109T043142Z.json, SHA-256 9e3689da66af95560034086d41f356d6cded71a130609e379fad68aef5c693d2, source 04e76b2aa73e66d1a5b06eff53b27783630a2634, with exclusive pre-run identity/configuration snapshot and independent exit-status receipt. Earlier artifacts/replays remain unchanged. No raw prompts, transcripts, credentials or workplace data are published.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: Typed approvals passed the two previously failing approval scenarios. The fresh run still passed only 20 of 21 because the eval rejected the exact gateway-family safety read required by the skill. That read-list correction makes the offline replay green, not the fresh result. Further frozen trials need new authorization; unseen and live validation remain absent.

Planning evaluation journal: November 9, 2026

The final authorized fresh trial passed 17/18 original scenarios and 3/3 holdouts, or 20/21. Both previously failing approval scenarios passed with typed approval operations. The six-additional-invocation ceiling is exhausted; no further model calls are scheduled. Planning repeatability and Grade A remain unestablished.

Approval checking now separates operations from explanation. An identity action declares its operation and identity type; a role action declares its operation, role and exact gateway principal. Each retains its exact approval scope. The universal input vocabulary covers assignment and removal, without identifying the operation or values required by a scenario. Those are derived from the public skill and supplied evidence. Regression tests reject removal/negated operations, user-assigned identities, broader/additional roles, foreign principals/scopes, extra properties, missing approvals and unapproved import entering the approved command plan. No sentence special cases were added. Historical sentence-format outputs and scores remain unchanged.

The sole final recorded failure was the alias grader rejecting az resource list scoped to the exact supplied subscription, RG and gateway name. The management skill requires same-name family preflight for name-selected operations. The proposed plan preserved the required alias read/update/read order, complete backend pool, callable and policies; no sibling mutation or runtime success claim occurred. This exact scoped read is now enumerated, with negative tests rejecting foreign subscription/RG/name, omitted name and resource mutation. No runtime skill change or oracle input was needed.

Offline replay of the immutable final outputs passes 21/21 behavior and 21/21 syntax after that read-list correction. All 23 regenerated input/configuration files remain byte-identical to the final trial's snapshot. The fresh score remains 20/21. This is a corrected comparison, not a new successful model run or three-trial streak.

Execution provenance: Waza 0.40.1, Copilot SDK executor, embedded runtime 1.0.105, GPT-6.1 Sol, one trial for each of 21 tasks. All tasks reached the model; syntax passed throughout, and no session setup errors were recorded. All 21 runs had complete deny-all records and zero tool calls/events/denials. Benchmark exit was 1; independent saved-result policy verification exited 0. Pre-run identity/configuration snapshot and post-run hashes retained frozen source/configuration and unchanged prior artifacts.

The actual session-local SDK startup checks confirmed hooks disabled, no installed plugins or user MCP servers, no managed startup policy keys and existing authentication. No global plugin changes, new login, Azure/data-plane or WorkIQ action calls occurred. This is not an OS filesystem, credential or network sandbox.

The final trial consumed 42.65535 AI credits, 21 premium requests/turns, 144,346 input tokens, 12,872 output tokens, 26,250 cache-read tokens and 118,033 cache-write tokens, over 189.511 seconds. Assistant-owned cumulative usage is 276.39610 credits, 141 premium requests/turns and 1,249.828 seconds reported suite duration, including the disclosed console-only invocation. The initial trial plus all six additional invocations have been used.

Validation after the final offline correction: 54 evaluation tests and 10 portable-plugin tests pass, plus Bash syntax, exact generated binding/input identity and diff checks. These are offline checks, not fresh quality executions.

The authorized correction/evaluation pass ends at its ceiling. No all-green fresh trial exists; three unchanged-candidate green trials remain unverified. The three holdouts were independently authored but inspected during development, not newly blind acceptance evidence. Live CLI/data-plane behavior, progressive plugin loading and matched CLI performance still require separate authorization. Additional model execution also requires a new ceiling.

Evidence retained locally: immutable result quality-combined-gpt-6.1-sol-20261109T045204Z.json, SHA-256 acf0159a136e694c6ecceb25ae894788ff3fd0766ecd02b13fa96252462822b0, frozen trial source ee7911062027ca086daf55b1cd2c95fbb233f05b, with exclusive pre-run identity/configuration snapshot and independent exit-status receipt. Replay is separately saved; earlier artifacts and replay history remain unchanged. No raw prompts, transcripts, credentials or workplace data are published.

@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: The first frozen run still failed bootstrap: it exposed both an identity-spelling comparison and an ambiguous single-deployment versus whole-account request. No import or access change was executed. Stop paid repeats, make the requested scope explicit, and normalize only equivalent identity spelling before restarting repeatability.

2026-11-09: first frozen repeatability trial

The first of the three newly authorized frozen trials passed 20/21: 17/18 original scenarios and 3/3 development-inspected holdouts. Syntax passed 21/21. Planning completion is still not established.

Waza 0.40.1 used the Copilot SDK, embedded runtime 1.0.105, GPT-6.1 Sol and one trial per task. All 21 tasks reached inference. Independent post-result verification found deny-all on all 21, zero tool calls/events/denials and complete counts. No setup failure was found. Benchmark exit was 1; policy-verifier exit was 0. Source/configuration and prior artifact hashes were unchanged. The isolated profile had hooks disabled, zero installed plugins/user MCP servers and existing authentication. This is not an OS filesystem, credential or network sandbox.

The sole failure was bootstrap, with two grader findings:

  • Identity was represented as systemAssigned rather than the accepted identity spelling.
  • The pending command proposed account-wide provider onboarding, while the grader requires selected-deployment model onboarding. The request itself does not explicitly select between these scopes.

The plan kept both access changes pending, bound them to the exact gateway principal and Foundry account, and explicitly disclosed the all-deployment import. No Azure command was executed. These findings do not establish an unsafe skill mutation. They also do not justify calling the fresh result a pass: the request/oracle scope and identity representation need correction without weakening approval checks.

Execution stopped on repeated bootstrap nonprogress rather than spending the other two frozen invocations. No grader, skill or generated input changed during or after the trial; no offline replay was used to relabel it. Three unchanged-candidate all-green trials remain unmet. Live CLI/data-plane and matched CLI performance evidence remain separately unauthorized.

This invocation used 42.78910 AI credits, 21 premium requests/turns, 144,361 input tokens, 13,002 output tokens, 26,250 cache-read tokens, 118,048 cache-write tokens, and 181.589 seconds reported suite duration. Cumulative assistant-owned usage is 319.18520 credits, 162 premium requests/turns, and 1,431.417 seconds reported duration, including the previously disclosed console-only invocation. The original initial-plus-six allowance is used; 1/3 newly authorized frozen invocations used, 2 unused.

Local provenance: immutable result quality-frozen-gpt-6.1-sol-20261109T051600Z.json; matching exclusive identity/configuration/status records; candidate 3f9c268fb673d5285c15bb843aecfc65352380fe; result SHA256 f8d48422a8ba6bf08aa22ec55324a21a9eeee6e532cd49d76c8cf186e4d3673c. No raw prompts, transcripts or credentials are published. PR #32 and its branch remain untouched.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: Bootstrap now explicitly requests one existing deployment, with whole-account import tested separately. Equivalent identity case is accepted without relaxing role, principal, operation or approval scope. These are offline corrections: the saved answer still addressed the old request, and no fresh pass or repeatability is claimed. Next, prepare the corrected candidate for three separately authorized frozen trials.

2026-11-09: bootstrap scope and identity contract corrected offline

The bootstrap input now explicitly limits optional onboarding to one selected existing deployment. The separate whole-account scenario continues to require explicit approval of both existing deployments and both external access changes. No default was broadened, and no paid run was made in this correction pass.

This changes the bootstrap request, not the expected import scope to fit a model answer. The saved frozen output answered the prior ambiguous request; it is not evidence that the new input passes. All 23 regenerated configuration/input files were compared to the frozen snapshot: only tasks/selected-rg-bootstrap.yaml changed.

Identity fact, action and principal fields now normalize case-only spellings of the same SystemAssigned value. CLI arguments, operation names, roles and resource IDs are not case-normalized. User-assigned, combined, absent and negated identities, broader roles, wrong principals/scopes, extra fields, missing approvals, other deployments and account-wide imports remain rejected. The relevant source references describe system-assigned identity as a distinct gateway identity; this comparison does not claim that arbitrary identity strings or CLI enum spellings are interchangeable.

All 18 original requests plus three development variants were reviewed in one offline pass for hidden import/resource scope and identity-enum assumptions. No further input-scope or enum correction was justified. An all-task regression checks exact request/evidence binding, absence of grader metadata from prompts, and oracle subscription/RG/gateway/Foundry-account/deployment targets. The approved bulk-import scenario is checked separately from the newly explicit single-deployment case.

Validation: 57 evaluation tests and 10 plugin tests pass, generated 21-task deny-all binding passes, pinned empty-sandbox guard/resource parity remains covered, Bash syntax and whitespace checks pass. These are offline checks; no new Waza, model, Azure or data-plane call was made.

A separate replay preserves the immutable output and its old request: 20/21 behavior and 21/21 syntax. Identity spelling no longer fails; the pending account-wide operation remains rejected. The recorded fresh 20/21 is unchanged. Prior artifacts, identities, snapshots and replay histories are preserved. No new fresh score is claimed.

The three historical holdout tasks are now labeled development variants in reader-facing documentation and authoring help. Compatibility filenames, task IDs and CLI flags remain unchanged. They were repeatedly executed and inspected during development; genuine held-out validation is absent.

Repeatability resets with this correction. Paid execution remains stopped, with two unused invocations from the last authorization. They cannot establish the required three consecutive all-green 21-task trials on this changed candidate. Before further spending approval: retain the passing offline tests and exact generated binding, recheck actual SDK startup isolation and narrow preflight, record fresh exclusive configuration/source identities, and authorize at least three unchanged-candidate one-trial invocations. Stop on setup/policy/evidence failures or repeated nonprogress. Live CLI/data-plane and matched CLI baselines still require separate authorization.

Evidence: evals/waza/cases.yaml, grade.py, test_planning.py, regenerated static bootstrap fixture, evaluation guides and authoring help. The distinct ignored replay is replay-frozen-identity-only-20261109T052900Z.json, source SHA256 f8d48422a8ba6bf08aa22ec55324a21a9eeee6e532cd49d76c8cf186e4d3673c. Raw prompts, transcripts and credentials are not published. PR #32 and native-stack metadata remain untouched.

@paulyuk

Copy link
Copy Markdown
Member Author

TL;DR: The corrected bootstrap scenario passed both fresh runs. The first run passed all 21 scenarios, but the next passed 19: the eval rejected a correctly scoped alias list and an equivalent composite-counter description. The alias plan also omitted a gateway-family safety check required by the skill, which the current grader does not independently enforce. Stop paid repeats, correct those checks offline, and test that safety omission before restarting consistency trials. One green run is not repeatability.

Corrected frozen trials: November 9, 2026

Both runs used the same corrected skill, grader, fixtures and generated configuration. All 21 scenarios reached inference in each run. No shared setup failure was found. These are fresh results, not rescored historical answers.

Evidence First corrected frozen trial Second unchanged trial
Original scenarios 18/18 pass 16/18 pass
Development variants 3/3 pass 3/3 pass
Overall behavior 21/21 pass 19/21 pass
Native syntax 21/21 pass 21/21 pass
Benchmark / independent policy exits 0 / 0 1 / 0
Deny-all policy records 21/21 complete 21/21 complete
Tool calls / events / denials 0 / 0 / 0 0 / 0 / 0
AI credits 42.66460 42.50285
Premium requests / turns 21 / 21 21 / 21
Input / output tokens 144,375 / 12,574 144,376 / 12,712
Cache-read / cache-write tokens 25,000 / 119,312 26,250 / 118,063
Reported suite duration 187.779 seconds 187.171 seconds

What still fails

Alias routing: the exact selected-alias read/update/read sequence, three-model pool, unrequested fallback, priorities, weights, policies and callable are preserved. The eval rejects additional model-alias list reads scoped to the selected subscription, resource group and gateway because that native read is absent from its explicit list. The proposed commands have valid recorded native syntax.

That comparison defect is not the only issue. The management skill also requires a scoped same-name gateway-family check before name-selected writes. This plan includes the typed gateway read but omits the family inventory. The current grader permits the family read but does not independently require it. Removing the rejected alias-list commands in an offline diagnostic makes the current grader pass even though this precaution is still missing. That diagnostic exposes a coverage gap; it is not a new score or evidence of full skill adherence.

Policy accounting: the answer reports tokens versus requests correctly, rejects a billing hard-cap claim, and calls the partition identity/ipAddress combination. The authoritative accounting reference specifies one counter per identity/IP combination. The eval accepts identity/ipAddress but not the longer equivalent label. Replacing only that field in an offline diagnostic isolates the comparison failure. The saved output was not changed, and this does not change the fresh 19/21 result.

These are two recorded semantic-grader failures, zero syntax failures and zero identified setup/tool-policy failures, with a separate demonstrated safety-preflight omission and grader-coverage gap in the alias plan. There was no Azure mutation, tool attempt, role grant or quota-bypass attempt.

What to attempt next

Review the scoped-read and composite-partition checks together offline, rather than add unconstrained wording exceptions. Preserve negatives for foreign subscription/RG/gateway, sibling writes, missing or reordered alias mutations, and separate rather than composite counters. Require and test the gateway-family precaution for name-selected writes. A comparison fix alone must not turn that missing precaution into a safe-plan pass.

Any skill/grader/fixture correction resets consistency evidence. The first corrected run was green; the next was not, so there is no three-trial streak. I did not spend the last authorized invocation: it cannot establish three consecutive green trials after these findings. Further paid execution is stopped, not scheduled as an automatic loop.

The three extra scenarios have been repeatedly inspected during development. Their historical holdout filenames remain for compatibility; they are development variants, not unseen validation. Genuine held-out evaluation, progressive plugin loading, live CLI/data-plane behavior and matched CLI performance remain absent. No Grade A claim is made.

Execution boundary and provenance

  • Waza 0.40.1, Copilot SDK executor, embedded runtime 1.0.105, GPT-6.1 Sol, one trial per task. No retry, other model, agent, workflow, login or global plugin change.
  • Before each invocation, the actual session-local SDK reported hooks disabled, zero installed plugins/user MCP servers, no managed startup keys and existing authentication. Generated deny-all binding and narrowly checked non-strict preflight passed. Preflight itself did not establish live policy enforcement.
  • Both invocations used the inert-tested canonical evals/waza/run_planning.sh, with exclusive pre-run result identities, snapshots and immutable input/prior-artifact hashes. Source/configuration and prior artifacts remained unchanged through each run. Saved-result verification ran independently even when the second benchmark exited nonzero.
  • Startup isolation and deny-all records are not an OS filesystem, credential or network sandbox.
  • The latest approval allowed three corrected frozen trials using two remaining invocations plus one additional. Two were executed, one is unused. Globally, the initial corrected invocation plus nine additional invocations have been used.
  • These two runs total 85.16745 AI credits, 42 premium requests/turns and 374.950 seconds. Assistant-owned cumulative usage is 404.35265 credits, 204 premium requests/turns and 1,806.367 seconds, including the earlier disclosed console-only invocation. These are reported units, not a verified currency charge or CLI-relative measurement.
  • Frozen candidate: 7c6b1f42b07a63974da98612f68bad4a3735aa14. Base: 7874ce7f7c0f0674402d55b7de7eefacf8c70562. Update plugin to latest CLI and RP #32 and native-stack metadata remain untouched.
  • First result: quality-frozen-gpt-6.1-sol-20261109T054941Z.json, SHA-256 6022d6dfbde0dc2f7b496c55ef08b328132964cb30adcdd8be8aa7ac38db2660.
  • Second result: quality-frozen-gpt-6.1-sol-20261109T055346Z.json, SHA-256 9635a3a67151015dcdcdf96cc8d3c41a0ef9e341227942794ea1ec14e0cddfe9.
  • Raw prompts, model outputs, transcripts, credentials and workplace data remain local and unpublished. The 57-test offline suite passed before these trials; no post-run rubric change or aggregate replay relabels their results.

Readable historical journal

The 14 existing evaluation updates now have human TL;DR openings, edited in place without deleting their detailed historical evidence. Two comments had displayed only local-file references; their prepared redacted findings are restored below the opening, with the original references retained in a publication note. Historical commands remain historical, not current rerun instructions. The separate manually written progress comment was left untouched.

Initial findings;
routing corrections;
planning contract;
preflight;
input wiring;
first complete run;
launch evidence stop;
context and comparisons;
scoped reads;
Foundry prerequisites;
approval wording stop;
typed approvals;
bootstrap ambiguity;
explicit scope correction.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ations

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Copy link
Copy Markdown
Member Author

TL;DR: The eval rejected valid scoped alias reads and a valid composite-counter description. Those comparisons are corrected. It also missed a real problem: the second trial's alias plan omitted the gateway-family check required before a name-selected write. The grader now requires that check, and the skill places the existing precaution next to execution. Next, validate this corrected candidate in fresh trials; old outputs cannot prove the revised guidance works.

What changed

  • Native alias lists are accepted only for the exact selected subscription, resource group and gateway. Foreign or omitted scope and mutation substitutions remain rejected.
  • The alias plan must inspect same-name gateway/service collisions before updating the alias. The family inspection and initial alias read may occur in either order, but both must precede the update, followed by alias read-back. An alias list, late family inspection or missing read-back cannot substitute for these prerequisites.
  • Composite accounting accepts the verified identity/ipAddress combination terminology as well as the existing equivalent representation. It also accepts an explicit composite object containing exactly the identity and IP dimensions. Independent counters, duplicate/extra dimensions, negation, extra properties and malformed values do not pass.
  • The management skill's existing family-preflight instruction moved from target resolution to the execution section, with a link from target resolution. It now explicitly distinguishes alias inventory from parent-family verification and pre-write inspection from post-write read-back. No scenario answer or evaluator label was added to the skill.

Offline evidence, not a new model trial

The two immutable GPT-6.1 Sol results retain their fresh scores: 21/21, then 19/21. Each ran the original 18 tasks plus three development variants once through Waza 0.40.1 and Copilot SDK/runtime 1.0.105. Both previously verified complete deny-all evidence, zero tool calls/events/denials and syntax 21/21, with no identified setup failure.

A distinct, field-only offline replay against the corrected grader passes 21/21, then 20/21 behavior, with syntax 21/21 for both. The second output still fails its missing gateway-family preflight. The comparison corrections therefore do not award a pass to the omitted safety check. These replay counts are not fresh scores, repeatability evidence or validation of the repositioned skill text.

The targeted offline suite passes 59 tests, and the plugin suite passes 10. Regression coverage includes missing/late family inspection, exact mutation/read-back ordering, scoped-list acceptance, foreign subscription/RG/gateway rejection, independent/extra counter dimensions, malformed objects, pinned path-guard parity and input/oracle isolation. The combined configuration was regenerated for GPT-6.1 Sol, one trial for each of 21 tasks; static binding still reports runtimePolicyVerified: false, as expected without execution.

Request and synthetic-evidence meaning did not change. Selected skill context changes because the precaution moved. Any grader/skill correction resets the repeatability streak. The repeatedly inspected three variants remain development evidence, not genuinely unseen holdouts.

Spending and next validation

No Waza invocation, model call, Azure/data-plane call, authentication flow or global plugin change occurred in this correction pass. Prior artifacts remain unchanged. The latest pair's historical usage remains 85.16745 AI credits, 42 premium requests/turns and 374.950 seconds; cumulative assistant-owned usage remains 404.35265 credits, 204 premium requests/turns and 1,806.367 seconds, including the disclosed console-only invocation.

Paid execution remains stopped. One previously authorized invocation is unused. The minimum planning-completion evidence is three fresh passing invocations of the same corrected candidate across all 18 originals and three development variants, with complete execution/policy evidence. That needs authorization for two more invocations in addition to the remaining one; another correction would reset the streak. Genuine held-out evaluation, live gateway behavior, progressive loading and matched CLI performance remain absent and separately scoped.

Earlier evaluation journals were revised in place with human TL;DR openings, preserving historical details. Their canonical links are indexed in the preceding journal.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paulyuk

Paul Yuknewicz (paulyuk) commented Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

TL;DR: The latest trial failed because the eval rejected valid scoped reads, not because it found unauthorized execution. Those comparisons are corrected. A subsequent offline audit found a different grader gap: bootstrap and approved Foundry create plans could omit the required family check. That prerequisite is now enforced without weakening scope or approvals. Paid trials are stopped at the user's request. Fresh validation would still be needed for repeatability; the saved trial remains 19/21, while the corrected offline replay is 21/21.

Fresh trial: 19/21, not repeatability

The frozen candidate passed 16/18 original scenarios and 3/3 development variants. Native syntax passed 21/21. All 21 tasks reached inference, with zero recorded session errors; the aggregate errors: 0 is not the sole basis for that conclusion.

The actual result records copilot-sdk, gpt-6.1-sol and one run per task. Waza was the verified pinned 0.40.1 binary. The actual SDK startup probe reported runtime 1.0.105/protocol 3, hooks disabled, zero installed plugins and user MCP servers, no managed startup keys and existing authentication. Runtime files were included in the freeze checks. The prepared source/configuration identity and prior artifact hashes were recorded before launch.

Post-result verification passed all 21 requested-deny-all records, with no missing policy/count evidence and zero tool calls, tool events or denials. Benchmark exit was 1; policy-verifier exit was 0. Frozen source and all recorded prior artifacts remained unchanged during execution. No Azure/data-plane call, new authentication flow or global plugin change occurred. This is not an OS filesystem/credential/network sandbox.

What failed, and what was corrected

  • Bootstrap: the proposed plan kept the exact approved gateway create, bounded readiness/read-back, exact Foundry prerequisite reads and single-deployment import pending both access approvals. The rejected extra command was a model inventory read restricted to the selected subscription, RG and gateway. That relevant registration check is now explicitly accepted; broader scope and moving import into the approved command list remain rejected.
  • Stale-sync assessment: the plan retained fresh sync --dry-run, exact provider model and alias dependency reads, denied discovery as incomplete, and no destructive execution. Its rejected extra command was the exact same-name gateway-family inventory required by the management recipe. It was absent from this scenario's supplemental list.

Instead of adding that parent read separately to each future failing case, the grader now derives only two common parent-preflight reads from a management task's supplied subscription/RG/gateway identity: fully typed gateway show --ids, and same-name resource-family inventory. This shared predicate is used by both ordered and phased plans. Other asset reads remain explicitly task-scoped. It does not permit foreign/omitted selectors, arbitrary queries, secret reads, write substitutions, runtime-only management discovery or replacement/reordering of required operations.

The alias scenario still independently requires its family check before update and read-back after update. Correcting optional read coverage does not remove that safety requirement. No skill, request or evidence change was needed for these two failures; no oracle answers were added to agent inputs.

Offline evidence and spending

A distinct field-only replay of this immutable output passes 21/21 behavior and 21/21 syntax under the corrected grader. That is not a new model score, validation of unseen inputs or an unchanged-candidate streak. The grader change resets repeatability. All 60 evaluation regressions and 10 plugin regressions pass; the combined configuration was regenerated and its static deny-all/task binding verified. Negative cases cover all management tasks' exact parent scope, omitted selectors, foreign families, query/write substitutions, strict alias preflight/order and runtime-only rejection.

This trial consumed 42.86510 AI credits, 21 premium requests/turns and 188.562 seconds. It recorded 144,773 input tokens, 12,675 output tokens, 25,000 cache-read tokens and 119,710 cache-write tokens.

The user explicitly authorized up to three frozen trials using one remaining invocation plus two additional. Only the first was used; the failure stopped this round, leaving two unspent. Globally, the initial corrected invocation plus ten additional invocations have been used. Cumulative assistant-owned usage is 447.21775 AI credits, 225 premium requests/turns and 1,994.929 seconds, including the previously disclosed console-only invocation. No subsequent paid trial has run.

After the correction, the user explicitly selected Stop paid trials. The two remaining invocations will not run automatically. The implementation corrections are committed; the planning-completion criterion remains unmet rather than being relabeled as complete.

Three consecutive passing unchanged-candidate trials are not achieved. The corrected grader needs three new green trials; two remaining invocations cannot establish that target. The three repeatedly inspected variants remain development scenarios, not genuine holdouts. Live gateway behavior, progressive loading and matched CLI performance remain absent and separately scoped.

Offline consistency audit after the stop decision

The audit covered all 21 tasks, including nine management scenarios and 47 read-comparison rows. It inspected 243 saved task/run records from 12 immutable results, including the operator artifact's setup failures. These are retained records, not 243 new inference calls. Historical command comparisons use the current rules as diagnostics only; the old bootstrap request previously had different import-scope meaning, so those comparisons do not turn historical outputs into fresh or current-input scores.

The two new coverage defects were in bootstrap and approved whole-account Foundry onboarding: family inventory was accepted as an optional read, but its absence did not independently fail the plan. The existing skill already requires that precaution before name-selected writes. The grader now uses the shared exact parent-read definitions for both equivalence and the pre-write prerequisite. Both scenarios explicitly include family inventory. A missing or late check fails even if a future scenario forgets to add it to its operation list. The alias check remains mandatory.

Equivalent flag order, quoting and verified short forms match without treating parent reads as child reads. A stronger typed gateway ID may match a name-selected gateway read only with the same explicit subscription/RG/name; the reverse cannot weaken a typed binding. Child reads and all required operations remain covered. Prerequisites precede writes and read-back follows; the approved Foundry identity/model read-backs can occur in either order. Foreign/omitted selectors, legacy-family IDs, extra queries or writes, missing operations, broader roles, foreign principals, negated/combined identities and independent counters remain rejected. Runtime-only scenarios do not gain management discovery.

Synthetic checks also found a false negative for repeating the exact already-required alias read before its update. The shared read predicate now accepts repetitions of approved, exactly scoped required reads, not arbitrary native GETs. An early read cannot satisfy a later required read-back, remove a prerequisite, authorize pending work or substitute for a write. This does not extend asset scope or secret/mutation permissions. An explicit empty behavior-command boundary does not inherit child reads from the historical command list.

All 63 offline evaluation regressions and 10 plugin regressions pass. A new distinct replay still passes 21/21 behavior, and the saved result's syntax and complete deny-all evidence remain 21/21 with zero activity. All 23 generated configuration/input files are byte-identical to the frozen snapshot. No skill text, scenario request, synthetic evidence, import scope or approval requirement changed; no oracle answers were added to inputs. The grader/oracle change resets repeatability again. Raw scores, costs and earlier replay history remain unchanged.

This suite evaluates application of supplied skill/reference/help text to fixed synthetic evidence. It does not test product skill discovery, selection or progressive loading, an OS sandbox, live Azure/data-plane effects or a matched CLI-performance baseline. Its CLI equivalence rules are bounded, not a complete interpreter for arbitrary queries/projections. Earlier saved plans genuinely omit family preflight; that remains a behavior failure, rather than another representation exception. No additional current skill defect was established by this audit. The earlier skill correction moved existing family-preflight guidance next to execution; this audit changes only evaluator coverage.

No Waza invocation, model call, Azure call or additional credit spend occurred during the audit. The user-selected stop remains in effect; both unused invocations remain unspent.

References

  • Frozen run candidate: a2b6624ceab82be5542982ecfebdfa66410e988d.
  • Immutable local result: quality-frozen-gpt-6.1-sol-20261109T062815Z.json; SHA-256 3e7040147a6b77f373128643102e33fa22a628661fd54bf21637f3f60687c69a. Raw result, configuration snapshot, identity/status receipts and distinct replay remain local and ignored.
  • Prior offline alias/accounting correction.
  • Historical journals, revised in place with human TL;DR openings.
  • Distinct local audit: offline-read-consistency-20261109T070220Z.json, containing artifact hashes and field-only comparison counts; raw outputs remain ignored and local.
  • Final shared-predicate audit and replay: offline-read-consistency-final-20261109T070953Z.json, with current grader/oracle hashes, all 12 unchanged artifact identities, 21/21 behavior/syntax replay and all 23 frozen input/configuration comparisons.
  • Offline audit correction commit: 157ec44fa118a8f2651a2c890d82faba80940912; this follow-up did not modify Update plugin to latest CLI and RP #32's branch or retarget Simplify AI Gateway skills and align portable plugin workflows #38.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@santiagxf

Copy link
Copy Markdown
Member

Feedback factored in to #55

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants